A Stochastic Search for Test Assembly, Item Pool Analysis, and Design (RR 04-01) (PDF)

Mixed integer programming (MIP) is a current approach for test assembly. Despite its popularity, it has two important disadvantages. Commercial MIP packages do not support all types of constraints and have a reluctance to go beyond the test assembly problem itself. A new test assembly algorithm based on a Monte Carlo random search is developed in the paper. The new approach attains the level of the performance of MIP tools, like CPLEX (ILOG, 2002), but handles nonlinear constraints as well. The nature of the approach allows us to address the following issues of pool analysis and design: compare strengths and weaknesses of different pools, identify the most restrictive constraint(s) for test assembly, and predict properties of the items that should be added to a pool in order to obtain more test forms. Computer experiments with operational pools provide possible answers to these problems. Introduction The problem of item pool design has been addressed by Boekkooi-Timminga (1991), Stocking and Swanson (1998), and van der Linden, Veldkamp, and Reese (2000). In the last paper, the authors used the mixed integer programming (MIP) approach to build a blueprint of an item pool providing a given number of test forms. The objective function minimized an estimated cost of item construction. Item categories were established and the desired number of items from each category was obtained from the MIP solution. One of the objectives of this paper is also to guide item construction, but a different approach is taken. The Monte Carlo approach identifies item types to add to an existing pool in order to acquire more test forms, and it assesses the quality of an existing item pool. Also, the effect of slight modifications to existing assembly constraints on pool usage is studied. The last 20 years have seen a widespread usage of automated test assembly at testing agencies. While most practical test assembly problems are NP-complete (Nemhauser & Wolsey, 1988), this does not mean that the assembly of a single linear test form is difficult. Most linear test assembly situations do not require the optimization of an objective function. Test specifications considered in the study include distributions on cognitive content, correct answer key, topics, word count, diversity, and mean score. Also, the section information and characteristic curves (Lord, 1980) must be within a range (–3, +3). Any combination of items meeting the specifications yields an acceptable test form. A typical item pool would give rise to a large number of ways to combine items to make a form (Theunissen, 1985; van der Linden, 1998; Armstrong, Jones, & Kunce, 1998). The problem can be formulated as an MIP problem (Nemhauser & Wolsey, 1988) and solved with a commercial software package (Armstrong & Belov, 2003). MIP is the most popular approach for test assembly (Fletcher, 2000), but it does, however, have important disadvantages. The problem becomes intractable when nonlinear constraints are enforced and violations of certain constraints must be analyzed. 1